Software programs are now used to analyze data and to perform countless other tasks in a variety of different fields. As these tasks have grown more complex, the complexity of the programs used to perform them has grown as well. Software programs routinely run into the thousands or millions of lines of code, and it is nearly impossible to manually eliminate all errors that might prevent a program from compiling or prevent the compiled code from operating as the developer(s) originally intended. In the general case, a software program will routinely contain many instruction sequences that perform specific activities needed for the program. A single software instruction sequence within a program may be very large, complex and difficult to debug, and the potential interactions between different software instruction sequences in a program can introduce further unanticipated behavior.
There are categories of software development support tools that have been developed to at least partially alleviate problems at the scale of an entire program. The category of debuggers enables software developers to interactively inspect the behavior of an executing program and instruction sequences in a development environment. The category of static analysis tools enables software developers to examine software source code in a development environment and search for common software errors that can be detected without executing the program. The category of system loggers produce text output from executing programs, where the content of the text was programmed into the source code by a software developer. System loggers are used in both a development and operational environment to provide a software developer with insight into the software behavior.
Debuggers allow software developers to interactively inspect and step through executing program code line by line to search for logic errors that result in unintended operation. Debuggers may also be used to identify a location in a program where a specific condition, or breakpoint, occurs, so that program and instruction sequence behavior can be interactively examined under known conditions. Debuggers are used by software developers in a development environment to troubleshoot a software problem, but are not typically useful for software executing in a production environment. Debuggers are highly intrusive tools because the software is stopped so that the internal logic and state can be inspected by a software developer.
Static analysis tools examine source code and search for flaws without executing the program itself. Static analysis tools analyze source code in a non-runtime environment to understand certain properties of the programs, such as potential memory leaks or unreachable code sections. Static analysis tools are similar to compilers in that the source code is processed to find certain classes of potential errors, but static analysis will not find most logic errors in software. Static analysis tools are useful in a software development environment, but are not routinely run on an operational system during program execution.
System loggers are routinely used to generate text output during software system operation. During software development, a software developer decides what information would be useful to log during operation and introduces source code into the program to interact with the logging service to write text or data to the log. The concept of logging from executing software is well-known to software developers, and the mechanism is widely used in both the initial development through deployment and operation of software systems.
Static analysis tools analyze source code to discover some types of common coding errors, and debuggers are helpful for inspecting programs and searching for flaws in their operation, but these categories of software development tools are generally unable to perform analysis on software in its ultimate operational environment. Given the complexity of modern computer programs, it may be unclear in a given end-user setting why a program is performing the way it is. A program routinely incorporates many software instruction sequences, and a software developer focuses attention on understanding and verifying the behavior of one instruction sequence at a time to demonstrate a correctly functioning program. Isolating the inputs, behavior and outputs of the instruction sequences within a program in an operational environment is not feasible with debuggers or static analysis tools. Furthermore, when a program is deployed in a setting with potentially unpredictable inputs and outputs, it may be desirable to know whether anomalous results generated by the program come from flaws in the instruction sequences or anomalies in the inputs. Conventionally, however, there is no suitable tool to examine instruction sequence execution behavior subsequent to and separate from the execution of the program in an operational environment. The predominant existing tool for operational support such as system loggers often intermix output from many different programs and instruction sequences. In addition, loggers may generate such voluminous quantities of system data that information about the behavior of a specific instruction sequence may be impossible to discern from the logs.